Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

build-support: Fix error when building images with many layers #141050

Merged
merged 1 commit into from
Oct 13, 2022
Merged

build-support: Fix error when building images with many layers #141050

merged 1 commit into from
Oct 13, 2022

Conversation

koenw
Copy link
Contributor

@koenw koenw commented Oct 9, 2021

When building a docker image using dockertools.buildLayeredImage, the resulting image layers are passed to jq through the command line. When building an image with too many layers this would exceed the maximum command line argument length.

Hence, we store the list of layers in the Nix store and pass them to jq as a file argument using --slurpfile.

Fixes #140908.

Motivation for this change

To fix #140908.

Things done
  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Build and loaded a docker image that failed previously
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • 21.11 Release Notes (or backporting 21.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

@koenw koenw requested a review from roberth as a code owner October 9, 2021 00:39
@ofborg ofborg bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux labels Oct 9, 2021
roberth
roberth requested changes Oct 9, 2021
View reviewed changes
@roberth roberth mentioned this pull request Oct 9, 2021
Closed
@koenw koenw requested a review from roberth October 9, 2021 21:36
roberth
roberth reviewed Oct 9, 2021
View reviewed changes
Copy link
Member

@roberth roberth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just some superficial things left.
Could you also squash the commit and prefix with dockerTools: instead? It's more recognizable.

@stale
Copy link

stale bot commented Apr 16, 2022

I marked this as stale due to inactivity. → More info

@stale stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Apr 16, 2022
@hlolli
Copy link
Member

hlolli commented Sep 23, 2022

I bumped into this jq Argument list issue, this PR fixed it! Do you want help getting this PR across the finishing line @koenw ?

@stale stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Sep 23, 2022
@koenw koenw force-pushed the dockertools_too_many_arguments branch from bba634b to 5fb1a00 Compare October 12, 2022 21:49
@koenw
Copy link
Contributor Author

koenw commented Oct 12, 2022

I bumped into this jq Argument list issue, this PR fixed it! Do you want help getting this PR across the finishing line @koenw ?

Thanks, I forgot about this PR :)

@roberth I implemented most of your suggestions (except one - see notes). Can you do a re-review please? :)

@roberth
Copy link
Member

roberth commented Oct 13, 2022 •
edited
Loading

@ofborg test docker-tools

@koenw
build-support: Fix error when building images with many layers
2f5fcda 
When building a docker image using `dockertools.buildLayeredImage`, the
resulting image layers are passed to `jq` through the command line. When
building an image with too many layers this would exceed the maximum
command line argument length.

Hence, we store the list of layers in the Nix store and pass them to
`jq` as a file argument using `--slurpfile`.

Fixes NixOS#140908.
@koenw koenw force-pushed the dockertools_too_many_arguments branch from 5fb1a00 to 2f5fcda Compare October 13, 2022 14:27
@roberth roberth merged commit 8fca2fd into NixOS:master Oct 13, 2022
@roberth
Copy link
Member

roberth commented Oct 13, 2022

Thank you @koenw!

@Janik-Haag Janik-Haag added the 12. first-time contribution This PR is the author's first one; please be gentle! label Jun 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@roberth roberth roberth approved these changes

Assignees
No one assigned
Labels
10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux 12. first-time contribution This PR is the author's first one; please be gentle!
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

dockertools.buildLayeredImage fails with jq: Argument list too long
4 participants
@koenw @hlolli @roberth @Janik-Haag